Summary

• Selection JSON schema now accepts a picks: [{id, confidence}] shape carrying per-pick confidence in [0.0, 1.0]. The legacy selected_section_ids shape still parses so older / weaker models keep working.
• When every confidence falls strictly below retrieval.abstain.below (default 0.4), /v1/query returns an abstention response (sections: [], abstained: true) and /v1/answer skips synthesis entirely and answers with a canonical refusal.
• Successful responses surface a confidences map keyed by section_id. Abstention responses additionally carry abstention_reason, min_confidence_threshold, and candidate_confidences.

Design rationale

• Additive on the strategy boundary. Result.SelectedIDs stays []tree.SectionID — Confidences is a separate map[SectionID]float64 field, omitted from JSON when empty. Callers that don't care about confidence see no API change.
• Strategies never abstain. Each strategy populates Result.Confidences if the model returned them; the abstention decision lives entirely in the API layer (internal/api/server.go). This keeps the strategies pure ("return what the model picked") and confines policy to one place.
• Abstention requires explicit signal. A nil confidence map (legacy LLM response, or new shape with no confidence keys populated) is the "no signal" sentinel. The abstention check returns false for nil / empty maps so older models cannot accidentally trip a refusal.
• "All picks below" semantics. If even one pick scored at-or-above the threshold, the engine has enough signal to surface evidence — abstention is reserved for the case where every candidate is weak. This matches the plan and avoids over-refusal.
• Trace-token absence on abstention. Replay isn't meaningful for an abstention (there's no retrieval result to reproduce), so abstention responses omit trace_token and aren't written to the replay store.

Opt-out / configuration

Test plan

• go build ./... clean
• go vet ./... clean
• go test ./... all green (all pre-existing tests pass + new coverage)
• New parse-side tests: new-shape, legacy-shape, mixed-shape, clamped, deduped, new-shape-no-confidences
• New strategy tests: SinglePass / ChunkedTree / Agentic surface Confidences, strategies never abstain
• New config tests: defaults, env overrides (enable / disable / parse / edge), bad-input rejection, validation
• New API tests: shouldAbstain predicate, helper sentinels, respondAbstained shape (query + answer), synthesis LLM tripwire (must not be called on abstention path), trace_token absent on abstention

Before / after examples

LLM new-shape response → confidences populated

// LLM returns:
{"picks":[{"id":"sec_a","confidence":0.82},{"id":"sec_b","confidence":0.31}],"reasoning":"x"}

// /v1/query response (no abstention because 0.82 ≥ 0.4):
{
  "document_id": "...",
  "sections": [{"id": "sec_a", ...}, {"id": "sec_b", ...}],
  "confidences": {"sec_a": 0.82, "sec_b": 0.31},
  ...
}

LLM all-low response → abstained

// LLM returns:
{"picks":[{"id":"sec_a","confidence":0.12},{"id":"sec_b","confidence":0.20}]}

// /v1/query response:
{
  "document_id": "...",
  "query": "...",
  "strategy": "chunked-tree",
  "sections": [],
  "abstained": true,
  "abstention_reason": "no candidate section scored above the confidence threshold",
  "min_confidence_threshold": 0.4,
  "candidate_confidences": {"sec_a": 0.12, "sec_b": 0.20}
}

// /v1/answer response (synthesis skipped):
{
  "document_id": "...",
  "answer": "I cannot answer this question from the supplied document.",
  "citations": [],
  "abstained": true,
  ...
}

Mixed-shape response handled

// LLM returns (some picks with confidence, some without):
{"picks":[{"id":"sec_a","confidence":0.9},{"id":"sec_b"},{"id":"sec_c","confidence":0.4}]}

// confidences map surfaces only present scores:
{"confidences": {"sec_a": 0.9, "sec_c": 0.4}, ...}
// sec_b is in sections[] but absent from confidences (no signal for that pick)

Legacy response → no abstention

// LLM returns:
{"selected_section_ids":["sec_a","sec_b"]}

// /v1/query response: NO confidences map, NO abstention check fires.
// Older models continue to work unchanged.

Summary by CodeRabbit

Release Notes

• New Features

  • Added confidence-driven abstention: when all candidate section confidences fall below a configured threshold, the API returns an abstention response with empty results instead of uncertain answers.
  • New enable_abstain parameter on query and answer endpoints for per-request abstention override control.
  • Responses now include per-section confidence scores for transparency.

• Configuration

  • New retrieval.abstain configuration block with enabled toggle and below confidence threshold (default: 0.4, range: 0.0–1.0).